Developing, evaluating and validating a scoring rubric for written case reports
نویسندگان
چکیده
OBJECTIVE The purpose of this study was to evaluate Family Medicine Clerkship students' writing skills using an anchored scoring rubric. In this study, we report on the assessment of a current scoring rubric (SR) used to grade written case description papers (CDP) for medical students, describe the development of a revised SR with examination of scoring consistency among faculty raters, and report on feedback from students regarding SR revisions and written CDP. METHODS Five faculty members scored a total of eighty-three written CDP using both the Original SR (OSR) and the Revised SR1 (RSR1) during the 2009-2010 academic years. RESULTS Overall increased faculty inter-rater reliability was obtained using the RSR1. Additionally, this subset analysis revealed that the five faculty using the Revised SR2 (RSR2) had a high measure of inter-rater reliability on their scoring of this subset of papers (as measured by intra-class correlation (ICC) with ICC = 0.93, p = 0.001. CONCLUSIONS Findings from this research have implications for medical education, by highlighting the importance of the assessment and development of reliable evaluation tools for medical student writing projects.
منابع مشابه
Exploring Novice Raters’ Textual Considerations in Independent and Negotiated Ratings
Educators often employ various training techniques to reduce raters’ subjectivity. Negotiation is a technique which can assist novice raters to co-construct a shared understanding of the writing assessment when rating collaboratively. There is little research, however, on rating behaviors of novice raters while employing negotiation techniques and the effect of negotiation on their understandin...
متن کاملRubric-based Automated Japanese Short-answer Scoring and Support System Applied to QALab-3
We have been developing an automated Japanese shortanswer scoring and support machine for the new National Center written test exams. Our approach is based on the fact that accurately recognizing textual entailment and/or synonymy has been almost impossible. The system generates automated scores on the basis of evaluation criteria or rubrics, and human raters revise them. The system determines ...
متن کاملDeveloping and Validating the Excellence Model for the Strategic Knowledge Management Case Study: Mobarakeh Steel Company
The aim of this study is to design the excellence in Mobarakeh Steel Company's strategic knowledge management. To achieve this goal, the researcher has used qualitative and quantitative methods. Therefore, the research method is the mixed one. This study, based on the nature of the data and data collection procedure, is a descriptive, exploratory research. It follows and applies development obj...
متن کاملWriting Analytics for Epistemic Features of Student Writing
Literacy, encompassing the ability to produce written outputs from the reading of multiple sources, is a key learning goal. Selecting information, and evaluating and integrating claims from potentially competing documents is a complex literacy task. Prior research exploring differing behaviours and their association to constructs such as epistemic cognition has used ‘multiple document processin...
متن کاملAutomated Scoring of a Summary-Writing Task Designed to Measure Reading Comprehension
We introduce a cognitive framework for measuring reading comprehension that includes the use of novel summary writing tasks. We derive NLP features from the holistic rubric used to score the summaries written by students for such tasks and use them to design a preliminary, automated scoring system. Our results show that the automated approach performs well on summaries written by students for t...
متن کامل